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Introduction 


The Child Care and Development Block Grant (CCDBG) Act of 


2014 and the Child Care and Development Fund (CCDF) Final This rejeteley is published as a part of 
the Initial Effects of Child Care 


Rule approved in 2016 outlined major policy changes in four 


areas: (1) health and safety, (2) consumer education, (3) equal Reauthorization on Child Care Markets 
access to high-quality care for children in families with low- project. The goal of the project is to 
incomes, and (4) quality improvement and support for the understand early effects and 
workforce..' The Consolidated Appropriations Act, 2018 implementation issues related to new 
approved a significant increase in CCDF funding to support policy requirements from the CCDBG 
the implementation of these changes. Act of 2014. 

CCDF has historically required lead agencies to spend a 

portion of their full CCDF award (including Discretionary, The project is led by MEF Associates in 
Mandatory, and Federal and State Matching funds) on quality | Partnership with Child Trends and is 
improvement activities (commonly referred to as the quality funded by the Office of Planning, 
set-aside). The Final Rule added provisions that gradually Research, and Evaluation (OPRE) in the 
increased the quality set-aside from four percent to nine Administration for Children and 
percent over a five-year period (fully effective by Fiscal Year Families (ACF) and the Office of the 
[FY] 2020) for states and territories. Based on an analysis of Assistant Secretary for Planning and 
FY 2012 expenditure data, 39 states and territories were Evaluation, U.S. Department of Health 


already spending over 9 percent on quality when the Final and Human Services (HHS). 
Rule was published.* Additionally, the Final Rule required 
states and territories to invest another three percent of their full CCDF award on improving the supply and 
quality of care for infants and toddlers (Commonly referred to as the infant-toddler set-aside) as of FY 
2017. 


The CCDBG Act of 2014 and the Final Rule required lead agencies to fund at least one of the following ten 
activities aimed at improving the quality of child care services (referred to as quality activities) ?: 


e Activity 1: Supporting the training, professional development, and postsecondary education of the 
child care workforce as part of a progression of professional development. Throughout this report, 
we refer to this activity as workforce training and PD. 


e Activity 2: Improving upon the development or implementation of the early learning and 
development guidelines. Throughout this report, we refer to this activity as early learning and 
development guidelines. 


e Activity 3: Developing, implementing, or enhancing a tiered quality rating and improvement system 
for child care providers and services to meet consumer education requirements. Throughout this 
report, we refer to the quality rating and improvement system portion of this activity as QRIS. 


e Activity 4: Improving the supply and quality of child care programs and services for infants and 
toddlers. Throughout this report, we refer to this activity as infant and toddler supply and quality. 


e Activity 5: Establishing or expanding a statewide system of child care resource and referral (CCR&R) 
services. Throughout this report, we refer to this activity as CCR&R. 


1 For more information on changes made in the CCDBG Act of 2014, see 
https://www.acf.hhs.gov/sites/default/files/documents/occ/ccdf final rule fact_sheet.pdf. 
2 Child Care and Development Fund Program, 45 C.F.R. 67567 (September 30, 2016). 

3 Child Care and Development Fund Program, 45 C.F.R. § 98.53 (September 30, 2016). 


e Activity 6: Facilitating compliance with lead agency requirements for inspection, monitoring, 
training, and health and safety standards. Throughout this report, we refer to this activity as 
compliance of health standards. 


e Activity 7: Evaluating and assessing the quality and effectiveness of child care programs and services 
offered, including evaluating how such programs positively impact children. Throughout this report, 
we refer to this activity as child care evaluation and assessment. 


e Activity 8: Supporting child care providers in the voluntary pursuit of accreditation by a national 
accrediting body with demonstrated, valid, and reliable program standards of high-quality. 
Throughout this report, we refer to this activity as supporting accreditation. 


e Activity 9: Supporting lead agency or local efforts to develop high-quality program standards 
relating to health, mental health, nutrition, physical activity, and physical development. Throughout 
this report, we refer to this activity as high-quality standards for health activities. 


e Activity 10: Carrying out other activities, including implementing consumer education provisions, 
determined by the lead agency to improve the quality of child care services provided, and for which 
measurement of outcomes related to improved provider preparedness, child safety, child well-being, 
or kindergarten entry is possible. Throughout this report, we refer to this activity as other activities. 


The CCDBG Act of 2014 also required states and territories to report how they invest in quality activities as 
well as how they measure progress to improve quality. In CCDF Plans,* states and territories prospectively 
report plans for investing in each of these quality activities for the next three-year period (planned 
investments). The plans only distinguish between investments that are covered by CCDF and non-CCDF 
funding. The Office of Child Care (OCC) reviews these plans as part of its process to determine state and 
territory compliance with the Final Rule. 


The Quality Progress Reports (QPR) is an annual report in which lead agencies provide information about 
how they used funding and measured progress for each of the quality activities to increase access to 
high-quality child care. In the QPR, states and territories retrospectively report whether they used any of 
three funding sources in the past fiscal year (i.e., CCDF quality set-aside funds, non-CCDF funds, or some 
of the increased CCDF funds from the Consolidated Appropriations Act, 2018) to support each of the 
quality activities(reported investments).° When states and territories report use and percentage of quality 
set-aside funds invested for each quality activity, the QPR does not ask lead agencies to distinguish 
between the quality set-aside and the infant-toddler set-aside except for when they report the funding 
percentages for infant and toddler supply and quality (activity 4). Thus, we assume that responses for all 
other quality activities may include both types of CCDF funds. 


States and territories also describe how they are measuring progress in each of the ten quality activities in 
the QPR. The QPRs act as status reports and are not used by OCC for determining compliance with the 
Final Rule.® Data from CCDF Plans and QPRs offer a glimpse into the degree to which planned 
investments match later reported investments and how states and territories prioritized investing in 
various quality activities (see Figure 1 for alignment of reported funding sources across federal reports). 


4 For more information about CCDF Plans, see https://www.acf.hhs.gov/occ/plans. 

> For more information about Quality Progress Reports, see https://www.acf.hhs.gov/occ/resource/ccdf-acf-pi-2019- 
09 

5 The Office of Child Care uses annual Financial Reports (ACF-696) to assess compliance with quality set-aside and 
infant-toddler set-aside percentages. Data from the FY 2018 Financial Reports (ACF-696) were not available at the 
time of analysis. 


Figure 1. Funding sources included in analyzed CCDF Plans and QPR 


CCDF funds 


Quality set-aside 
(including infant- 
toddler set-aside 
unless otherwise Expectation to use | Non-CCDF funds 


Any (unspecified) increased funds (unspecified) 


This report aims to answer the following questions, summarize findings from analyses conducted in 
November 2019, highlight gaps in the available information on quality investments, and offer implications 
for future research. 


1) What were states’ and territories’ planned and reported investments in quality activities? 


2) How did states and territories report measuring progress to improve quality? 


State and Territory Quality 
Investments Analysis 


Data sources, analysis, and findings are presented separately for each of the two questions. 


Question 1. What were states’ and territories’ planned and 
reported investments in quality activities? 


Data Sources and Elements 


To understand states’ and territories’ approaches to investing in at least one of ten quality activities, we 
analyzed information reported by states and territories about their plans to invest and their reported 
investments from three time periods (Figure 2). First, we analyzed data from CCDF Plans to understand 
states and territories’ plans to invest in quality activities. 


e Fiscal Year 2016-2018 CCDF Plan. In September 2015, states and territories provided plans for 
administering CCDF between October 1, 2015 and September 30, 2018. They reported whether they 
planned to invest in eight of ten quality activities.’ These data illustrated states’ and territories’ 
planned investments in quality immediately following reauthorization. These responses were coded 
as “Yes” (the lead agency planned to invest CCDF or non-CCDF funds) or “No” (the lead agency did 
not plan to invest CCDF or non-CCDF funds). 


7 States and territories responded in Section 7.1.2 where they indicated whether they would be investing in each 
quality activity. The FY 2016-2018 CCDF Plan did not ask about workforce training and PD (activity 1) and early 
learning and development guidelines (activity 2). 


e Fiscal Year 2019-2021 CCDF Plan. In September 2018, states and territories provided plans for 
administering CCDF between October 1, 2018 and September 30, 2021. They reported whether they 
planned to invest in ten quality activities over the three-year period.® These responses were coded 
as “Yes” (the lead agency planned to invest CCDF or non-CCDF funds) or “No” (the lead agency did 
not plan to invest CCDF or non-CCDF funds). 


Next, we analyzed data from Quality Progress Reports (QPR) to compare states’ and territories’ reported 
investments with their planned investments. 


e Fiscal Year 2018 QPR. In May 2019,? states and territories reported in the FY 2018 QPR about 
activities conducted between October 1, 2017 and September 30, 2018. These data described 
reported investments in quality for each of the ten quality activities.’° States reported whether they 
used CCDF quality set-aside funds (and the estimated percentage of set-aside funds spent), and 
increased CCDF funds from the Consolidated Appropriations Act 2018 and/or non-CCDF funds for 
each of the ten quality activities. These responses were coded as either “Yes” (the lead agency used 
this funding source) or “No” (the lead agency did not use this funding source), or “Not Applicable.”." 


Figure 2. Timeline of CCDBG milestones and reporting requirements 


CCDBG Act of 2014 __ Final Rule published Expected implementation 
November 2014 September 2016 September 2018 


FY 2016-2018 CCDF Plan 
Oct. 1, 2015 — Sept. 30, 2018 


FY 2018 QPR 
Oct. 1, 2017 — Sept. 30, 2018 


FY 2019-2021 CCDF Plan 
Oct. 1, 2018 — Sept. 30, 2021 


Analysis 


By examining financial investments, we can understand how states and territories aimed to improve the 
quality of the child care system through the types of activities they financially supported. 


Because states and territories can blend funding sources to support quality activities, we first determined 
whether states or territories made any investments in the quality activities, regardless of funding source 
(Table 1). In the FY 2016-2018 and the FY 2019-2021 CCDF Plan, we calculated the number and 
percentage of states and territories that responded “Yes” to the question about planning to invest either 


8 States and territories responded in Section 7.2.1 Use of Quality Funds about whether they used “any CCDF funds” or 
“non-CCDF funds” for each quality activity in the FY 2019-2021 CCDF Plan. 

° Lead agencies were given additional time to complete the FY 2018 QPR. For more information on guidance offered 
to lead agencies for submission, see https://www.acf.hhs.gov/occ/resource/ccdf-acf-pi-2019-01. 

10 States and territories responded in Sections 2.10, 3.2, 4.12, 5.7, 6.3, 6.4, 7.5, 8.3, 9.2, 10.2, and 11.2 about whether 
the “state or territory [spent] CCDF quality set aside funds; non-CCDF funds; or expected to use at least some of the 
increased CCDF funds from the Consolidated Appropriations Act, 2018.” 

"1 For some quality activities, such as enhancing a tiered QRIS, responses may be coded as “Not Applicable” if the 
state or territory did not have a QRIS. 


CCDF funds or non-CCDF funds in each quality activity. In the FY 2018 QPR, we calculated the number and 
percentage of states and territories that responded “Yes” to the question about reported investments 
across any of the three funding sources (i.e., quality set-aside, increased CCDF funds, or non-CCDF 
funds)..'* Then, we compared the number and percentage of states that planned to invest and whether 
they reported investing in each of the ten quality activities (Table 2). 


Next, we analyzed states’ and territories’ reported investments in each of the ten quality activities by 
funding source from the FY 2018 QPR to understand how states were blending reported funding sources 
(Table 3 and 4). When states and territories indicated that they used CCDF quality set-aside funding, they 
were also asked on the QPR to report the percentage of CCDF quality set-aside funding that was invested 
in that particular quality activity (Table 5).'? We developed five ranges of percent investment of the CCDF 
quality set-aside and reported the number of states with investments in each range (e.g., 1-10 percent) for 
each quality activity..4 


Findings 


We summarize the findings from our analysis of planned and reported investments in quality activities by 
whether states and territories invested in each quality activity regardless of which reported funding source 
they used as well as by individual or combinations of funding sources. 


Investment in quality activities using any source of reported funding 


Examining whether states and territories planned or reported investments in each of the ten quality 
activities allows us to understand how they approached improving quality for CCDF activities. We provide 
the number and percentage of states that indicated they planned to invest funds (i.e., CCDF or non-CCDF) 
to support any of the ten quality activities from the FY 2016-2018 and FY 2019-2021 CCDF Plans as well as 
reported investments from the 2018 QPR (Table 1). 


12 We excluded states and territories that responded “Not Applicable” across each of the three funding sources for a 
specific quality activity from the denominator. 

13 We worked closely with the Office of Child Care to clarify the percentages provided by states and territories. There 
were some discrepancies and limitations. For example, some states indicated that they used the quality set-aside but 
were unable to report the percentage of investment. These missing data are not reported here. In some instances, we 
used a percentage that differed from what was reported because the state or territory provided more information in 
the description that clarified the number. For example, one state noted that they reported a proportion of funds 
rather than the percentage of funds (0.001” as a proportion would convert to 0.1% but as a percentage would be 
0.001%). In these cases, we converted the proportion to a percentage before analyzing. Finally, we coded some data 
as missing because it was likely reported incorrectly (e.g., a response of “4425” without any additional clarification). 

14 The five categories of percent investment ranges were selected to identify meaningful ranges. We applied the same 
range categories for each of the ten quality activities. 


Table 1. Number and percentage of states and territories investing in quality activities using any 
source of reported funding (CCDF and non-CCDF funds) 


Planned 
investments in 
this activity in the 
FY 2016-2018 


Reported 
investments in 
this activity in the 


Planned 
investments in 
this activity in the 
FY 2019-2021 


CCDF Plan FY 2018 QPR CCDF Plan 

Quality Activity (n=56) 1° (n=55).1¢ (n=56).17 
Activity 1: Workforce training and PD No data 55 (100%) 56 (100%) 
pail ee ae and No data 33 of 54 (61%) 48 (86%) 
Activity 3: QRIS A8 (86%) 52 of 53 (98%) 51 (91%) 
ae igs and toddler supply 52 (93%) AT (85%) 56 (100%) 
Activity 5: CCR&R services 38 (68%) 36 of 53 (68%) 43 (77%) 
ne Compliance of health 50 (89%) 53 (96%) 52 (93%) 
sees care evaluation and 41 (73%) 43 (78%) AT (84%) 
Activity 8: Supporting accreditation 30 (54%) 26 of 54 (48%) 30 (54%) 
Activity 9: High-quality standards for 

Smee ; . nee aaa) ao Te) 
Activity 10: Other activities 29 (52%) 38 of 53 (72%) 25 (45%) 


Source: FY 2016-2018 CCDF Plan; FY 2018 QPR; and FY 2019-2021 CCDF Plan 


States and territories reported multiple investments in quality activities, though prioritized investments in 
some activities over others. Findings indicate that: 


e Over time, almost all states and territories planned to make investments and then reported 
investments in workforce training and PD (activity 1),1 QRIS (activity 3), infant and toddler supply 
and quality (activity 4), and compliance of health standards (activity 6). 


e The fewest number of states and territories reported investing in supporting accreditation (activity 8) 
with just about half of them planning to and then reporting investments in this quality activity. 


15 States and territories were considered as planning to invest if they reported “Yes” to using CCDF and/or non-CCDF 
funds. The FY 2016-2018 CCDF plan did not ask states and territories to report plans to invest for activity 1 and 2. 

16 One territory did not report investments in the FY 2018 QPR. States and territories were considered to have made 
investments if they reported “Yes” to investing CCDF quality set-aside funds, increased CCDF funding from the 
Consolidated Appropriations Act, 2018, and/or non-CCDF funds. When a state or territory responded “Not 
Applicable” across the three reported funding sources, we excluded them from the denominator. The denominator for 


this column is 55 unless otherwise stated. 


17 We considered states and territories as planning to invest if they reported “Yes” to using CCDF and/or non-CCDF 


funds. 


18 There was no information about plans for investment for this activity in the FY 2016-2018 CCDF plan. 


e Over time, more states and territories made plans to invest in quality activities. Comparing plans to 
invest in quality activities from the FY 2016-2018 CCDF Plan with the FY 2019-2021 CCDF Plan, the 
number of states and territories planning to invest increases across almost all quality activities..1 


We also wanted to examine whether states and territories that planned to invest as of the FY 2016-2018 
CCDF Plan reported investments during the last year of the three-year period (Table 2). We examined 
whether states and territories reported “Yes,” that they would invest in an activity in their FY 2016-18 
CCDF Plan, and then whether they reported “Yes” to making investments in the FY 2018 QPR. We also 
noted whether they invested even if they did not indicate that they were planning to do so. 


Table 2. Number and percentage of states and territories comparing planned investments and 
reported investments using any source of reported funding 


Planned to invest in the FY Did not plan to invest in the FY 
2016-2018 CCDF Plan and 2016-2018 CCDF Plan but 
reported investments in the FY | reported investments in the FY 
2018 QPR”° 2018 QPR”' 

Activity 1: Workforce training siondlats nie iaia 

and PD 

Activity 2: Early learning and Ne.dai ioaae 


development guidelines 


Activity 3: QRIS 47 of 48 (98%) 5 of 8 (63%) 


Activity 4: Infant and toddler 


; 43 of 52 (83%) 4 of 4 (100%) 
supply and quality 
Activity 5: CCR&R services 28 of 38 (74%) 8 of 18 (44%) 
fee Compliance of health AT of 50 (94%) 6 of 6 (100%) 
des ie ncaa aa chaenc 33 of 41 (80%) 10 of 15 (67%) 
ie rea 21 of 30 (70%) 5 of 26 (19%) 
Activity 9: High-quality standards 28 of 40 (70%) 8 of 16 (50%) 


for health activities 


Activity 10: Other activities 24 of 29 (83%) 14 of 27 (52%) 


Source: FY 2016-2018 CCDF Plan and FY 2018 QPR 
States and territories seemed to invest similarly as they planned in their CCDF Plan. 


e Across all activities for which there is data, over 70 percent of states and territories that planned to 
invest in their 2016-2018 plan reported investing funds from any reported funding source. Because 
states and territories could have used another funding source not captured in the FY 2018 QPR to 
invest in these quality activities, we cannot conclude that they did not make any investments in 


19 Because states and territories were not able to report on plans to invest in workforce training and PD (activity 1) 
and early learning and development guidelines (activity 2), they may have included their investments in these 
activities in other activities (activity 10). 

20 This is a percentage of those that reported in the FY 2016-2018 CCDF Plan that they were planning on make an 
investment in this quality activity. 

21 This is a percentage of those that reported in the FY 2016-2018 CCDF Plan that they were not planning on making 
an investment in this quality activity. 


these activities. Additionally, some states and territories invest in activities even when they did not 
plan to invest. 22 


e Nearly all states and territories planned and reported investments in more than one quality activity 
even though they are only mandated to invest in one quality activity according to reports in the FY 
2018 QPR.”? Six states or territories planned to invest in all eight quality activities in the FY 2016- 
2018 CCDF Plan, and six reported investments in all 10 quality activities in the FY 2018 QPR.*° Of 
the six states that reported investments, three were the same states that reported plans to invest in 
all eight quality activities in the FY 2016-2018 CCDF plan. In the FY 2019-2021 CCDF plans, ten states 
and territories indicated that they plan to invest in all ten quality activities. 


Investment in quality activities by reported funding source 


Next, we examined how states and territories used CCDF and non-CCDF funds in each of the quality 
activities in the FY 2018 QPR. For CCDF funds, the QPR asked states and territories about their use of the 
CCDF quality set-aside (which may have included the infant-toddler set-aside unless otherwise noted) and 
their expected use of the increase in CCDF from the Consolidated Appropriations Act, 2018. The QPR also 
asked whether any non-CCDF funds were used to support the quality activities. 


Table 3 includes the number and percentage of states that responded “Yes” for each funding source 
separately in the QPR. The QPR did not ask whether states used CCDF dollars in addition to the quality 
set-aside or increased funds to invest in each activity. Thus, we cannot make conclusions about the extent 
to which states and territories used any CCDF dollars to support these quality activities. Table 4 includes 
the number and percentage of states that responded “Yes” to different combinations of funding sources 
in the QPR. 


Table 3. Number and percentage of states and their FY 2018 QPR reported investments by funding 
source 


CCDF quality set- 

aside funds CCDF increased Non-CCDF funds 
Quality Activity (n=55) funds (n=55) (n=55) 
Activity 1: Workforce training and PD 53 (96%) 34 (62%) 36 (66%) 
Activity 2: Early learning and ‘ a ‘ 
development guidelines sek icles eeu SHES) 
Activity 3: QRIS 47 (85%) 34 (62%) 28 (51%) 
ie, 4: Infant and toddler supply and 25 (45%) 35 (64%) 19 (35%) 
Activity 5: CCR&R services 31 (56%) 14 (25%) 14 (25%) 
Geen Compliance of health AT (85%) 30 (55%) 16 (29%) 
Activity 7: Child care evaluation and 29 (53%) 19 (35%) 22 (40%) 
assessment 
Activity 8: Supporting accreditation 20 (36%) 8 (15%) 6 (11%) 


22 There were three activities where more states and territories made investments in FY 2018 than the number that 
planned based on reports from the FY 2016-2018 CCDF Plan (Activity 3, 6, and 10). 

23 In the FY 2016-2018 CCDF Plan, only one state indicated that they planned to invest in one quality activity during 
the two-year period. This state, however, reported investing in nine quality activities in the FY 2018 QPR. 

4 Note: States and territories only responded to eight of ten quality activities. 

25 Note: Reported investments only included investments from CCDF quality-set aside funds, non-CCDF funds, or 
expectations for increased CCDF funds from the Consolidated Appropriations Act, 2018. 


CCDF quality set- 

aside funds CCDF increased Non-CCDF funds 
Quality Activity (n=55) funds (n=55) (n=55) 
Activity 9: High-quality standards for . 6 ‘ 
health activities 23 (42%) 17 (31%) 21 (38%) 
Activity 10: Other activities 26 (47%) 19 (35%) 20 (36%) 


Source: FY 2018 QPR 


Table 4. Number and percentage of states that invested combinations of reported funding sources 


in quality activities in FY 2018 


Reported using both 


Reported using at CCDF funding 

least one CCDF Reported using both | sources and non- 

funding source CCDF funding CCDF funds 
Quality Activity (n=55) 26 sources (n=55) 2” (n=55) 28 
Activity 1: Workforce training . a S 
sat Dn 54 (98%) 33 (60%) 21 (38%) 
Activity 2: Early learning and . a e 
development guidelines oot”) re) ne 
Activity 3: QRIS 51 (93%) 30 (55%) 15 (27%) 
Activity 4: Infant and toddler 43 (78%) 17 (31%) 9 (16%) 
supply and quality 
Activity 5: CCR&R services 34 (62%) 11 (20%) 4 (7%) 
Activity 6: Compliance of health é 5 . 
eandarde 52 (95%) 25 (45%) 9 (16%) 
Activity 7: Child care evaluation 35 (64%) 13 (24%) 5 (9%) 
and assessment 
Activity 8: Supporting 5 5 5 
accreditation sic Sails pas en) 
Activity 9: High-quality . é 6s 
standards for health activities S02) nie) eae) 
Activity 10: Other activities 33 (60%) 12 (22%) 4 (7%) 


Source: FY 2018 QPR 


States and territories reported utilizing multiple funding sources to invest in quality activities. Across 
quality activities, the number of states and territories that used CCDF funds was greater than the number 


that invested non-CCDF funds. 


e When looking at the number of states and territories that used at least one of the two CCDF 
funding sources reported in the QPR to invest in a quality activity, at least three quarters of states 
and territories reported investing CCDF funds for workforce training and PD (activity 1), QRIS 
(activity 3), infant and toddler supply and quality (activity 4), and compliance of health standards 


26 States and territories were counted if they used CCDF quality set-aside funds and/or expected to use increased 


CCDF funds from the Consolidated Appropriations Act, 2018. 


27 States and territories were counted if they used CCDF quality set-aside funds and expected to use increased CCDF 
funds from the Consolidated Appropriations Act, 2018. 
28 States and territories were counted if they used CCDF quality set-aside funds, expected to use increased CCDF 
funds from the Consolidated Appropriations Act, 2018, and used non-CCDF funds. 


(activity 6). Half or more of states and territories utilized both sources of CCDF funding to support 
workforce training and PD (activity 1) and QRIS (activity 3). 


e Few states and territories reported investing CCDF funds for early learning and development 
guidelines (activity 2) and supporting accreditation (activity 8). 


e Half or more of the states and territories reported using non-CCDF funds for workforce training and 
PD (activity 1) as well as QRIS (activity 3). Few states and territories used non-CCDF funds to support 


accreditation (activity 8). 


e Some states used CCDF and non-CCDF funds to support quality activities, with workforce training 
and PD (activity 1) and QRIS (activity 3) being the top two activities that were supported with this 


combination of funding. 


When states and territories reported using CCDF quality set-aside funds, they also reported the 


percentage of the CCDF quality set-aside used for each of the ten quality activities. We provide the 
number (and percentage) of states and territories whose investment percentage fell into five investment 


ranges (Table 5). 


Table 5. State/territory quality set-aside investment for each quality activity, by investment range 


Above Above Above 

1% to 10% to 25% to Above 
Quality Activit Range <1% 10% 25% 50% 50% 

yi y 9g 

— i ig raining 1 <1%-71% | 4 (8%) | 14 (27%) | 16.31%) | 14(27%) | 3.6%) 
Activity 2: Early learning and 
development guidelines <1%-11% 8 (44%) 9 (50%) 1 (6%) 0 (0%) 0 (0%) 
(n=18) 
Activity 3: QRIS (n=45) <1%-100% 2 (4%) 15 (33%) | 17 (38%) | 6 (13%) 5 (11%) 
Activity 4: Infant t | 
eee ee <1%-50% | 2(8%) | 11 (46%) | 8 (33%) | 3(13%) | 0 (0%) 
Aa ie aaa <1%-67% | 3(10%) | 10 (32%) | 13 (42%) | 13%) | 4 (13%) 
Activity 6: Compliance of ’ . . 5 é 6 6 
health standardsaned7) <1%-100% 7 (15%) 12 (26%) | 15 (32%) | 8 (17%) 5 (11%) 
Activity 7: Child care 
evaluation and assessment 1%-100% 4 (15%) 14 (52%) | 4(15%) 1 (4%) 4 (15%) 
(n=27) 
Activity 8: Supporting 7 ene - 5 5 5 5 
accreditation m=17) <1%-100% | 10 (59%) | 6 (35%) 0 (0%) 0 (0%) 1 (6%) 
Activity 9: High-quality 
standards for health activities 1%-76% 0 (0%) 16 (70%) | 3 (13%) 2 (9%) 2 (9%) 
(n=23) 
pa 10: Other activities | <4%-100% | 4(17%) | 13(54%) | 2(8%) | 2(8%) | 3(13%) 


Source: FY 2018 QPR 


29 States and territories were instructed to not include the required three percent of infant-toddler set-aside in this 


report. 


10 


States and territories varied widely in the percentage of CCDF quality set-aside used to invest in each of 
the quality activities. Key findings are highlighted below: 


e States and territories reported the lowest percentage of CCDF quality set-aside investments for early 
learning and development (activity 2) and supporting accreditation (activity 8). 


e States and territories reported investing most of their funds on the suggested quality activities 
(activities 1-9), with over 70% of states and territories spending 10% or less on other quality 
activities (activity 10). 


e Five states or territories reported spending over half of their quality set-aside on QRIS (activity 3) 
and compliance of health standards (activity 6). 


e Fifteen percent or fewer of states and territories that reported using CCDF quality set-aside funds for 
quality activities spent more than 50 percent of their quality set-aside on any single activity. This 
suggests that they seem to invest funds in multiple activities to support quality, rather than focusing 
on investing most of their funds into one activity. 


Potential Gaps and Future Implications 


The current analysis uncovered various gaps that have implications for future efforts to analyze quality 
investments. 


Because states and territories are asked to report quality investments for different purposes across 
CCDF reports, it is important to understand how these data do or do not align before analyzing 
them. Each report allows OCC to gather different aspects of information about lead agency investments. 
OCC uses CCDF Plans for compliance purposes and asks states and territories to report broadly about 
their plans to use any CCDF funds to invest in ten quality activities. The QPR asks states and territories to 
report on specific funding sources (i.e., CCDF quality set-aside, increased CCDF, and non-CCDF) to gather 
more detailed information on the type of funding used to support these quality activities. The QPR does 
not distinguish between the quality set-aside and the infant-toddler set-aside (except in the spending 
section of activity 4). OCC does not use the QPR information to monitor for compliance. Finally, OCC asks 
states and territories to provide expenditure information multiple times a year in financial reports (ACF- 
696), which requests separate reports of the amount spent on the quality set-aside and infant-toddler set- 
aside each quarter. It does not, however, ask states to distinguish how they used increased funding from 
the Consolidated Appropriations Act, 2018. These differences in purposes and questions mean that the 
information is not necessarily comparable across reports. Thus, researchers cannot perfectly compare data 
from one report to another because the way lead agencies report on various funding sources differs 
across reports (Figure 2). If understanding how states and territories invest in quality activities with specific 
funding sources across time becomes important, it will be useful to explore approaches for gathering 
similar information across sources over time. 


Variability in how states and territories report investments makes it more difficult to interpret and 
analyze investments. States and territories varied in how they reported the percentage of the CCDF 
quality set-aside used to invest in each of the ten quality activities in the FY 2018 QPR. This variation is 
likely because lead agencies are not required to report their investments in a particular format for the 
QPR. We note below three types of variability that made it challenging to understand the data for this 
analysis: 


1. Proportion vs. percentage. When asked for the percentage of the CCDF quality set-aside used to 
invest in each activity, some states seemed to have reported a proportion of their funds instead of a 
percentage. For example, a response of .01 could be interpreted as .01 percent if reported as a 
percentage or 1 percent if interpreted as a proportion. The instructions asked for a percentage, so we 
typically reported the number as a percentage. In a few instances, though, we realized—from reading 
additional text offered by states and territories to explain the percentage—that the number was a 
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proportion. In those instances, we adjusted the data. Researchers may want to be aware of the 
potential variation in how lead agencies report information. Future work could explore the feasibility 
and usefulness of providing additional guidance, examples, or supports to help lead agencies report 
the requested information more consistently. 


2. Over- or under-counting investments. A total percentage greater than 100 percent might be 
possible if states and territories counted an investment, such as funding used for infant and toddler 
provider training, in two or more activities (e.g., workforce training and PD [activity 1] and infant and 
toddler supply and quality [activity 3]). When states and territories report the percentage of the CCDF 
quality set-aside funds used to invest in each quality activity, researchers should understand that the 
sum of a state or territory's percentages across the ten quality activities may not equal 100 percent of 
their funds. 


3. Inability to report. A few states could not disaggregate funding to identify the percentage of CCDF 
quality set-aside invested. To understand the percentage of the funds used, states and territories 
might need additional support to adjust how they track the allocation of these funds. We 
acknowledge that this might require adjustments to state or territory reporting mechanisms, which 
could be expensive or time consuming. 


Additional research is needed to understand the effectiveness of various approaches to 
investments (e.g., funding more vs. fewer quality activities). More states and territories reported 
investing in quality activities over time. For example, states and territories may have decreased the 
amount they invested in individual quality activities to support new activities if the total amount of 
funding toward quality activities did not increase. Analyzing data from future QPRs could help to track the 
size of investments over time, coupled with examining the total amount of funding used for quality set- 
aside funding in financial reports (ACF-696). 


Future research could provide more in-depth information about state and territory investments in 
quality improvement as well as how CCDF administrators make investment decisions. The QPR does 
not ask states and territories about how leaders determined which funding source to use or how they 
allocated funds across activities. Interviews with CCDF administrators and their teams could address 
questions about how states determine whether to invest in particular quality activities, which funding 
sources they use and why, how they decide whether to invest in more versus fewer quality activities, and 
how they gather information about the effectiveness of their investments. 


CCDF Plans and QPRs provide data about intentions to invest and percent of funds invested. 
Expenditure data would help round out the information. Expenditure data from financial reports (ACF- 
696) could be used, along with QPR data, to calculate the investments for each quality activity (by 
multiplying the percentage invested in each activity on the QPR with the total amount of quality set-aside 
expended). Because states and territories may not have distinguished between the quality set-aside and 
infant-toddler set-aside in the QPR, however, it would be important to combine the total amount 
expended for both the quality set-aside and the infant-toddler set-aside from the financial reports (ACF- 
696). 


Long-term analysis of investment data would strengthen our understanding of state and territory 
quality improvement investments. Analyzing trends in data could address questions like, “Are states 
and territories investing similarly over time? Are there different investment profiles (e.g., some states and 
territories invest a little across many activities whereas others invest more heavily in a few)?” 
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Question 2. How are states measuring progress In 
improving quality? 


Data Sources and Elements 


Fiscal Year 2018 Quality Progress Report (QPR). States reported about activities conducted between 
October 1, 2017 and September 30, 2018 in the FY 2018 QPR. To learn more about how states and 
territories measure progress on quality investments, we analyzed responses to the item asking state and 
territories to “describe the measures used and progress made during October 1 to September 30 of the 
last federal fiscal year. Include examples and numeric targets where possible.” Examining these 
approaches offers insight into how states and territories are monitoring their progress and can help to 
inform future efforts to understand progress in improving child care quality. 


Analysis 


To understand how states and territories are measuring progress in improving quality, we coded 
responses across the four quality activities in which most states invested (see Table 2). 


e Activity 1: Workforce training and PD 
e Activity 3: QRIS 
e Activity 4: Infant and toddler supply and quality 
e Activity 6: Compliance with health standards 
We coded the following in each state or territory response: 
e The number of progress measures mentioned 
e Whether the response included numbers (e.g., 225 providers, 595 facilities, 4000 children) 


e Whether the response included data about change (i-e., whether the measures increased or 
decreased over time) 


We also noted what was measured (e.g., number of trainings offered, percent of programs that improved 
their QRIS rating). 


Findings 


Nearly all (95 percent) states and territories responded to the QPR question about how they measure 
progress on their various quality activities. More than 90 percent of states and territories described at 
least one measure, and a few included several. Some of the measures were more directly relevant to the 
activity than others, and some but not all measures included data about change over time (Table 6). 


13 


Table 6. How states and territories measure progress in four quality activities 


Described at Described Average Noted 
least 1 multiple number of Range of change over 
measure measures measures measures time 
n (%) n (%) described reported n (%) 
eel Age 52 (95%) 43 (83%) 42 1-24 21 (40%) 
Activity 3: QRIS (n=53) 48 (91%) 34 (71%) 2.3 1-9 23 (48%) 
Activity 4: Infant and 
toddler supply and 52 (96%) 44 (85%) 4.0 1-23 20 (38%) 
quality (n=54) 
Activity 6: Compliance 
of health standards 50 (93%) 42 (84%) 3.5 1-17 11 (22%) 
(n=54) 


Source: FY 2018 QPR 


Of the states and territories that described at least one measure, 72-81 percent of them included a 
number (e.g., number of families served or number of trainings offered) *°. A small group of states and 
territories described their measurement approach in general but did not provide numbers. Some of these 
responses described progress that could not be quantified (e.g., developing a data system). In other cases, 
the responses were general descriptions of the measure (e.g., monitor the number of providers who 
complete infant and toddler trainings, use of pre- and post-tests to measure change in knowledge from 
trainings) without the specific numbers (e.g., 225 providers completed training). Although these general 
responses typically do not provide information about the extent to which progress was made, they do 
provide information about how states and territories are measuring progress. 


States and territories often described multiple measures, with an average of two to five measures per 
activity among states and territories using more than one measure. Table 7 includes examples of 
responses for each activity type. Many of these measures were basic counts or outputs (e.g., number of 
providers completing training, number of programs participating in QRIS, number of trainings for infant 
and toddler teachers, number of inspections completed). Not all measures were closely related to the 
activity topic. For instance, some measures included in the infant and toddler supply and quality activity 
were not specific to infants and toddlers (e.g., number of providers who completed training on early 
learning and development standards, number of trainings). 


30 The QPR asked for descriptions of measures used and progress made for each quality activity. It also requested 
“examples and numeric targets where possible.” In reviewing the QPR responses, we believe that the numbers 
represent descriptions of measures and progress made rather than targets set because most statements were written 
in past tense (e.g., 5,548 hours of training completed, 36 providers earned their CDA). Thus, we use the term 
“number(s)” throughout this report. It is possible, though, that the information was intended to be a numeric target. 
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Table 7. Examples of measure descriptions 


Examples of Measure Descriptions 


Activity 1: Workforce 
training & PD 


Trainings, training participants, TA visits, coaching hours, individuals with 
degrees or credentials, programs receiving TA, programs at upper tiers of QRIS 


Activity 3: QRIS 


Programs participating in QRIS, programs at each QRIS rating level, slots in QRIS 
programs, slots in highly rated programs, programs increasing their rating level, 
programs maintaining their rating level 


Activity 4: Infant and 
toddler supply and 
quality 


Trainings, training participants, TA contacts, professional development sessions 
completed, infants and toddlers served in high quality programs 


Activity 6: 
Compliance of 
health standards 


Inspections, new licenses, health and safety trainings, trainings completed, 
enforcement actions; capacity of licensed programs 


Source: FY 2018 QPR 


As noted in Table 6, fewer states and territories provided information to document changes over time (i.e., 
progress). About half of state and territory responses for measuring progress for QRIS (activity 3) 
described changes over time (e.g., the number or percent of programs at the highest rating tiers). A little 
over one third of responses for workforce training and PD (activity 1) and infant and toddler supply and 
quality (activity 4) provided information about changes over time. Table 8 provides examples of the 
responses that described change over time. These responses typically included numbers for the current 
and previous year. 


Table 8. Examples of measure descriptions that examined change over time 


Examples of Measure Descriptions for Change Over Time 


Activity 1: Workforce 
training & PD 


Course or trainings offered, staff receiving scholarships, degrees completed, 
staff receiving educational scholarships, CCDF programs that are high quality (as 
measured by QRIS) 


Activity 3: QRIS 


Programs participating in QRIS, programs with higher tier ratings, programs 
receiving subsidies that participate in QRIS 


Activity 4: Infant and 
toddler supply and 
quality 


Infants/toddlers enrolled in high quality programs, infants/toddlers receiving 
subsidies who are enrolled in high-quality programs, percent of high-quality 
programs serving infants and toddlers, knowledge gained in infant and toddler 
training 


Activity 6: 
Compliance of 
health standards 


Capacity of licensed programs, licensing trainings completed, licensor caseloads 


Source: FY 2018 QPR 


Potential Gaps and Future Research Implications 


States and territories varied in their responses about how they measure progress in the various 
quality activities, which limited analysis. Responses varied in terms of the number of measures 
included, the level of detail provided, and the inclusion of information to note progress or change over 
time. This variation may limit the type of analysis that can be completed when understanding quality 


investments. 


The variation in progress measures used across states and territories made it more challenging to 
summarize information across states. If states and territories consistently reported at least one similar 
measure for some activities, then it would have been easier to monitor progress over time across states. 
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Although it is not possible to require common metrics, it would be useful to explore whether there are a 
few good measures that states and territories could consider tracking over time for each of the ten quality 
activities. With the current data, researchers interested in summarizing information across states would 
need to develop themes or categories and then code state and territory responses into the categories. 


States and territories might benefit from guidance and technical assistance on various topics 
related to measuring progress: 


Approaches to measuring progress on the various quality activities. The FY 2018 QPR responses 
suggest that states are tracking multiple data elements, some of which are less relevant to the 
activity topic. While some counts are appropriate for measuring progress (e.g., number and 
percentage of programs participating in QRIS, number and percentage of the workforce with a 
college degree), other counts may be less useful to document progress. For instance, knowing that 
the number of trainings offered increases every year suggests that training has expanded over time 
and may reach more providers. However, it does not provide any information about the extent to 
which providers participate in the trainings and find them useful or whether those trainings improve 
provider knowledge, practices, or interactions with children. CCDF administrators and their teams 
might benefit from supports to review and refine the measures they use to document progress in 
improving quality. Selecting and measuring one or a few indicators over time for each quality 
activity that are closely aligned to the success of the activity may be less burdensome and also yield 
better data to inform CCDF administrators and their teams. 


Suggested approaches for evaluating their quality investments. Although many of the measures 
described in the QPR provide some basic information about quality improvement, CCDF 
administrators and their teams may need to ask different questions and gather other data to better 
understand the impact of their investments. To understand the extent to which the quality activities 
are implemented as intended and the effectiveness of investments in improving quality, CCDF 
administrators and their teams should consider answering various questions, such as: Are the 
investments supporting evidence-based strategies? Are those strategies implemented as planned? 
Are they reaching the targeted providers? Are they improving interactions and practices that 
support children’s learning and development? What are the costs and benefits of various 
investments? What is the relationship between investments in quality and outcomes of interest? 


In addition to gathering information about individual quality investments or activities, state and 
territory leaders may also have questions about how multiple investments are working together to 
support quality. For instance, if states and territories invest in five core strategies to support the 
supply and quality of infant and toddler care, then it would be useful to evaluate each of the five 
strategies (e.g., is a particular infant and toddler coaching program improving practices among its 
provider participants) as well as the extent to which all of the investments are working together to 
improve the supply and quality of infant and toddler care (e.g., percent of licensed programs serving 
infants and toddlers, percent of programs serving infants and toddlers that participate in QRIS). A 
logic model." could help guide evaluations by showing how investments or strategies are intended 
to affect goals. 


31 A logic model is a graphic representation of how the elements or inputs of a program, intervention, or strategy 
relate to or are proposed to change short-term and long-term outcomes. From Vogt, W. P. & Johnson, R. B. (2011). 
Dictionary of statistics and methodology: A nontechnical guide for the social sciences, 4th edition. Thousand Oaks, CA: 
SAGE Publications Inc. 
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Conclusion 


This report highlights states’ and territories’ initial investments in quality activities as well as their 
approaches to measuring the progress of improving quality across these activities. The data suggest that 
more states and territories invested in quality activities over time. Access to increased CCDF funds may 
have offered states and territories opportunities to invest in quality activities. Additionally, states and 
territories may have implemented policy changes required by reauthorization between the submission of 
the FY 2016-2018 CCDF Plan and the FY 2019-2021 CCDF Plan that allowed them to focus more on quality 
activities in future years. For example, they may have used the initial years after reauthorization to 
implement health and safety training that, once in place, did not require as much financial investment. 
Thus, they may have invested some of those dollars in other activities. An examination of expenditures in 
financial reports (ACF-696) will help document how investments in quality activities change across years. 
Findings also indicate that states and territories varied in the type of funding source they used and the 
percentage of each funding source they allocated to various quality activities. Additionally, they varied in 
their approaches to measuring progress in improving quality across each activity. 


Stakeholders may be interested in understanding how states and territories are investing in quality 
improvement initiatives, especially those outlined by the reauthorization of the CCDBG Act of 2014. CCDF 
reports that gather information about how states invest in quality activities (e.g., CCDF Plans, QPR, or 
financial reports [ACF-696]) can be a rich source of information, though the information across reports 
does not always align because the reports were designed for different purposes and collect different 
information. Each report contains useful pieces of information about state and territory investments that 
vary in terms of the content, level of specificity, and timing. Researchers using these data should note the 
limitations of the report data and acknowledge the limitations in analyzing information across data 
sources. 


The COVID-19 pandemic hit the U.S. during the data analysis portion of this project. This report describes 
quality investments early after reauthorization and before the COVID-19 pandemic. State and territory 
investments will likely change to address new or shifting priorities. Examining data from future CCDF 
Plans, QPRs, and financial reports (ACF-696) will help the field understand how investments changed after 
the pandemic began. New data (e.g., adding new questions to these reports *2, collecting new data) may 
also be useful to understand what, how, and why states and territories changed their quality investments. 


32 OCC is revising federal reports in response to the COVID-19 pandemic. 


17 


